Multi-Image Deblurring

ABSTRACT

Embodiments of the invention describe a method for reducing a blur in an image of a scene. First, we acquire a set of images of the scene, wherein each image in the set of images includes an object having a blur associated with a point spread function (PSF) forming a set of point spread functions (PSFs), wherein the set of PSFs is suitable for null-filling operation. Next, we invert jointly the set of images and the set of PSFs to produce an output image having a reduced blur.

FIELD OF THE INVENTION

This invention related generally to image processing, and moreparticularly to removing motion blur effects in images acquired of amoving object.

BACKGROUND OF THE INVENTION

Motion blurs result from relative motion between the camera and thescene while an image is acquired. Motion blurred images can be restoredup to lost spatial frequencies by image deconvolution, provided that themotion is shift-invariant, at least locally, and that a blur function,also known as a point spread function (PSF), that caused the blur isknown.

Maintaining invertible motion PSF is not possible in conventionalimages. A box function due to a finite exposure time corresponds to aconvolution with a low pass filter, and hence a frequency transform ofthe PSF contains zeros (nulls). The frequencies corresponding to thenulls of the PSF are lost, which makes the deblurring ineffective.Conventional methods use specialized cameras to determine the PSF.

For example, one conventional method opens and closes a shutter duringan exposure time using a broadband binary code. The broadband code doesnot have any nulls in the frequency domain, thereby making the resultingPSF invertible. However, that method requires specialized hardware,assumes a constant background, and requires a manual PSF estimation andobject segmentation.

A motion invariant imaging method moves the camera with a constantacceleration while acquiring the image. The key idea is to make themotion PSF invariant to object velocity within a certain range. Thismakes segmentation and PSF estimation unnecessary. However, that methodrequires a prior knowledge of the direction of the motion, createsartifacts at object boundaries due to occluding background, andcritically introduces blur even in the static parts of the scene.

A wavefront coding method uses a cubic phase plate in front of the lensto make the PSF invariant to scene depths. However, that method resultsin defocus blur on scene parts originally in focus.

Another method open and closes the shutter of the camera with abroadband binary code to make the PSF invertible. Accelerating cameramotion makes the motion PSF invariant to the velocity of the object, atthe cost of blurring static parts.

Conventional consumer cameras perform image stabilization using adaptiveoptical elements controlled by inertial sensors to compensate for cameramotion.

A hybrid Camera uses a hybrid imaging system that estimates the PSFusing an auxiliary low-resolution high frame rate to deblur the highresolution primary sensor images. However, that method requires anauxiliary camera for PSF estimation.

Motion PSF has been estimated by combining partial information fromsuccessive images having two different exposures: a short exposure forPSF estimation and a long exposure for an image deblurring using theestimated PSF. However, a special camera is required for acquiring theimage with a short exposure.

Multiple co-located cameras with overlapped exposure time andreconfigurable multi-camera array have also been used to increase thetemporal resolution of the acquired images. However, it is desired touse a single conventional camera for PSF inversion.

It is therefore desired to deblur of a scene even if the PSF of eachimage is non-invertible due to a blur, and the images are acquired by asingle conventional camera.

SUMMARY OF THE INVENTION

It is an object of subject invention to invert motion blur in a set ofimages even if the point-spread function (PSF) of each image isnon-invertible due to a blur.

Blurred images exhibit nulls (zeros) in the frequency transform of thePSF, leading to an ill-posed deconvolution. Hardware solutions to avoidthis problem require specialized devices such as a coded exposure cameraor an accelerating sensor motion. We use conventional video cameras andintroduce the notion of null-filling and joint-invertability of multiplePSFs. The key realization is to acquire a set of images of the scenewith varying PSFs, so that nulls in the frequency component of one imagecan be filled by other images. The combined frequency transform becomesnull-free, making deblurring well-posed.

Embodiments of the invention describe a method for reducing a blur in animage of a scene. First, we acquire a set of images of the scene,wherein each image in the set of images includes an object having anassociated blur and point spread function (PSF) forming a set of pointspread functions (PSFs), wherein the set of PSFs is jointly-suitable fornull-filling operation. Next, we invert jointly the set of images andthe set of PSFs to produce an output image having a reduced blur.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a method for multi-image deblurringaccording embodiments of invention;

FIG. 2 is a schematic of an ill-posed point spread function (PSF)according the embodiments of the invention;

FIG. 3 is a schematic of PSFs null-filling according the embodiments ofthe invention;

FIG. 4 is a block diagram of a single image deblurring according theembodiments of the invention;

FIG. 5 is a block diagram of a combined linear system for multi-imagedebluring according the embodiments of the invention; and

FIG. 6 is a block diagram of a method for automatic deblurring accordingthe embodiments of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 shows a method 100 for multi-image deblurring of a set of images110 of a scene according to embodiments of out invention. Each image111-113 includes an object having a blur associated with a set 120 ofpoint spread function (PSF) 121-123. The method inverts jointly 140 theset of images using the set of PSFs to produce an output image 150having a reduced blur. The steps of the method are performed by aprocessor 101.

The images 110 are acquire by a camera such that the set of PSFs issuitable for a null-filling operation 130. The suitability is achievedby acquiring the images with different exposure times.

For example, in one embodiment, a first image 111 is acquired with afirst exposure time, and a second image 112 is acquired with a secondexposure time that is not an integer multiple of the first exposuretime. In some embodiments, we use a conventional camera having an autoexposure bracketing (AEB). AEB enables the camera to take a sequence ofimages at different exposures.

As shown in FIG. 2, for a single image 111 acquired by the camera with aconventional exposure, the motion PSF is a box filter 220. The discreteFourier transform (DFT) 230 of the PSF is a sine cardinal (sinc)function 240, which contains zeros 250, thus making the deblurringill-posed. The set of PSFs 120 includes a first PSF 121, a value of thefirst PSF at a particular frequency 250 equals zero, and a second PSF122, wherein a value of the second PSF at the particular frequency doesnot equal zero.

The embodiments of the invention use a novel concept of a PSFnull-filling. FIG. 3 shows that by combining multiple images 110acquired with different exposure times 320, the nulls in each individualPSF 330 are filled with data from other PSF. Accordingly, the combinedPSF 340 makes the debluring 140 well-conditioned.

Joint Invertability of Non-Invertible PSFs

Let f denote a sharp image of a scene including an object. We take Nblurred images i_(k) of the object, wherein each image i_(k) has adifferent PSF h_(k) forming a set of PSFs

i _(k) =f*h _(k) +n _(k) , k1 . . . N,   (1)

where * is a convolution operator and n_(k) is zero mean additive whiteGaussian noise with variance σ_(k) ². Let T_(k) be an exposure time forthe k^(th) image. We denote the Fourier transform of quantities usingcapital letters. The Fourier transform F(w), where w is a frequency, ofthe images I(w) is

I _(k)(w)=F(w)H _(k)(w)+N _(k)(w) k=1 . . . N.   (2)

If one-dimensional object motion is parallel to a sensor plane withconstant velocity, then each of the PSFs correspond to a box filter 220whose length 225 is proportional to the exposure time T_(k). Let r_(k)be the blur size in the k^(th) frame. Then

h _(k)(x)=1/r _(k) 0<x<r _(k).   (3)

Single image deblurring (SID) of any individual image is

$\begin{matrix}{{{F(w)} = {{{I_{k}(w)}{V_{k}(w)}} = {\frac{I_{k}(w)}{H_{k}(w)} = {{F(w)} + \frac{N_{k}(w)}{H_{k}(w)}}}}},} & (4)\end{matrix}$

where F(w) denote the Fourier transform of the deblurred image and

${V_{k}(w)} = {\frac{1}{H_{k}(w)} = \frac{H_{k}^{*}(w)}{{{H_{k}(w)}}^{2}}}$

is a Fourier transform of a corresponding deconvolution filter v_(k).

FIG. 3 shows a method for multi-image deblurring (MID), e.g., using Nimages 110. In the preferred embodiment, an optimal deconvolutionfilters V_(k)(w) is obtained by minimizing the noise power in the outputdeblurred image by

$\sum\limits_{k = 1}^{N}{{N_{k}^{2}(w)}{{V_{k}(w)}}^{2}}$

at each frequency w. Note that

${\sum\limits_{k = 1}^{N}{{V_{k}(w)}{H_{k}(w)}}} = 1$

to recover the sharp image.

Using Lagrange multiplier, the cost function is:

$\begin{matrix}{{{J(w)} = {{\sum\limits_{k = 1}^{N}{{N_{k}^{2}(w)}{{V_{k}(w)}}^{2}}} + {{\lambda( {{\sum\limits_{k = 1}^{N}{{V_{k}(w)}{H_{k}(w)}}} - 1} )}.}}},} & (5)\end{matrix}$

accordingly:

$\begin{matrix}{{{V_{k}(w)} = \frac{{H_{k}^{*}(w)}/{N_{k}^{2}(w)}}{\sum\limits_{k = 1}^{N}{{{H_{k}(w)}}^{2}/{N_{k}^{2}(w)}}}},} & (6) \\{{F(w)} = {{\sum\limits_{k = 1}^{N}{{I_{k}(w)}{V_{k}(w)}}} = {{F(w)} + {\frac{\sum\limits_{k = 1}^{N}{{H_{k}^{*}(w)}/{N_{k}(w)}}}{\sum\limits_{k = 1}^{N}{{{H_{k}(w)}}^{2}/{N_{k}^{2}(w)}}}.}}}} & (7)\end{matrix}$

If there are common zeros among all the PSFs 330 at a particularfrequency w 350, then H_(k)(w)=0 for all k at that frequency and V(w)becomes unstable.

If there are no common zeros in the Fourier transform of the PSFs, thenthe information lost in each individual image is acquired by some otherimage. The zeros in each individual PSF are filled by other PSFs.

Thus, if the set of PSFs does not have common zeros, then the combineddeconvolution can be made well-posed, even though each PSF isnon-invertible, i.e., the set of PSF is suitable for null-filling.

In one embodiment, the PSF is a motion PSF. However, other types of PSFare used by the embodiments. For motion PSF, this requires that theexposure times should not be integer multiples of each other

${P(w)} = \sqrt{\sum\limits_{k = 1}^{N}{{{H_{k}(w)}}^{2}/{N_{k}^{2}(w)}}}$

is an operator for combined deconvolution.

Multi-Image Deblurring

We formulate the motion blur as a motion smear matrix multiplied by thesharp image. As shown on FIG. 4, for a single image deblurring (SID),the images 111-113 are recovered with multiplication of motion smearmatrices 411-413 by the sharp image 150. However, if the PSF isill-conditioned, then the deblurring is not satisfactory because themotion smear matrix is determined by the PSF.

FIG. 5 shows a combined linear system for multi-image deblurring (MID),in which we combine the matrices 510 and the images 520. The singularvalues of motion blur matrices show that the combined deblurring systemA_(c) is better conditioned.

The convolution equation in the discrete domain is i_(k)=A_(k)f+n_(k)for each motion line, where A_(k) is a circulant motion smear matrix forthe image k, i_(k) is a vector describing a blurred object, f is avector describing a sharp object, and n_(k) is a vector describing noiseintensities along each motion line. For SID, a vector {circumflex over(f)} describing a deblurred object is obtained by minimizing the costfunction J=(i_(k)−A_(k)f)^(T)(i_(k)−A_(k)f) resulting in

{circumflex over (f)}=(A _(k) ^(T) A _(k))⁻¹ A _(k) ^(T) i _(k).   (8)

Similarly, for MID, the combined linear equation is

$\begin{matrix}{\begin{bmatrix}i_{1} \\\vdots \\i_{k}\end{bmatrix} = {{{\begin{bmatrix}A_{1} \\\vdots \\A_{k}\end{bmatrix}f} + \begin{bmatrix}n_{1} \\\vdots \\n_{k}\end{bmatrix}} = {{A_{c}f} + {n_{c}.}}}} & (9)\end{matrix}$

In Equation (9), A_(c) is a combined covariance matrix, and n_(c) is anoise variance.

In some embodiments, the estimated deblurred vector {circumflex over(f)} is obtained by minimizing the cost function

$\begin{matrix}{{{J = {{n_{c}^{T}n_{c}} = {\sum\limits_{k = 1}^{N}{( {i_{k} - {A_{k}f}} )^{T}( {i_{k} - {A_{k}f}} )}}}},{{which}\mspace{14mu} {yield}}}{\hat{f} = {( {\sum\limits_{k = 1}^{N}{A_{k}^{T}A_{k}}} )^{- 1}{( {\sum\limits_{k = 1}^{N}{A_{k}^{T}i_{k}}} ).}}}} & (10)\end{matrix}$

Accordingly, we invert jointly the set of images and the set of PSFsusing Equation (10) to produce an output image having a reduced blur.

Exposure Sequence Optimization

In one embodiment, we search for the exposure times to maximize theminimum of the combined operator P. In another embodiment, weincorporate sensor noise characteristics to account for signal dependentnoise. Since the variance of the electrons generated by photons linearlyincreases with the measured signal, the exposure time, σ_(k) ² is givenby σ_(gray) ²+βT_(k), where σ_(gray) ² is the dark noise and β is acamera dependent constant.

Using these parameters, we obtain the optimal exposure sequence byminimizing the decrease in signal-to-noise ratio (SNR) given by nf. Forcoded exposure, the search space is of the order of 2^(n), where n isthe code length, e.g. 52. The number of unknowns for MID is equal to thenumber of different exposure time used. Typically, three or fourdifferent exposure times are sufficient for deblurring, and thus thesearch is relatively small.

Automatic Deblurring

FIG. 6 shows a block diagram of the automatic deblurring method.Embodiments of the invention use joint PSF invertability for deblurringthe images 610 having an object moving in front of a non-smoothbackground.

PSF Estimation

For spatially invariant blur, PSF estimation 620 is represented as themultiplication of the image-space object velocity v and the exposuretime for each image. Object velocity is the ratio between an inter-imagemotion vector and an inter-image time lapse. For the spatially invariantblur, the inter-image motion vector is computed by matchingcorresponding image patches. However, different exposure times lead todifferent sizes of blur; and thus to facilitate matching and PSFestimation we repeat the exposure sequence for acquiring the images.Thus, every N^(th) image in the set of images 610 has the same exposure,where N is the number of different exposures used (≈3-4). Motion vectorscan be computed by matching the images acquired using the same exposure.Averaging the motion vectors for different exposures gives the finalestimate of the PSF.

Initialization

Let m_(k)(x,y) be the binary mask for the object in the k^(th) image andT_(i) be the inter-frame time. If b(x,y) is the background image withoutthe object, then the acquired motion blurred images i_(k) are given by

i _(k)=(f·m _(k))*h _(k)+(1−h _(k) *m _(k))·b.   (17)

We first estimate the background b. In some embodiments, the objectmoves sufficiently fast, i.e., each background pixel is occluded by theobject in less than 50% the images. Therefore, we use a median filteringalong a temporal direction. For slow moving objects, we use thebackground subtraction to determine the background b.

A blurred image has contributions from both the blurred foreground andthe background. The image blurring Equation (17) can be written in termsof an alpha matting equation as

i _(k) =αg+(1−α)·b,   (18)

where

$g = \frac{( {f \cdot m_{k}} )*h_{k}}{h_{k}*m_{k}}$

and α_(k)=h_(k)*m_(k). Deblurring of alpha maps can recover the binarysegmentation mask m_(k). Matting is typically used for non-opaque staticobjects, and we assume that the foreground motion blurred object isopaque and in sharp focus. Thus, the alpha map depends only on themotion blur and the matting foreground actually corresponds to theblurred object and not to the sharp object.

To compute initial alpha maps 630, we first generate a crude trimap foreach image by thresholding the difference between the input image i_(k)and the background image b. The trimap is 1 for the interior of themoving object, and is 0 for a background and unknown for the blurredregion. Morphological operations, such as hole-filling are applied toreduce noise. Using the trimap, alpha matting is performed on each frameindependently.

The blurred foreground f_(k) ^(b) is obtained by removing the backgroundcontribution 640 from each input image as

f _(k) ^(b) =i _(k)−(1−α)·b.   (19)

For spatially invariant motion blur, the alpha map is a ramp along eachmotion line and noisy.

Multi-Image Deblurring (MID)

Since the object appears at different locations in successive images,the blurred foreground f_(k) ^(b) needs to be aligned beforedeconvolution. For linear constant motion, the alignment corresponds toa shift in the image plane. Since we have the object velocity v, theshift between i^(th) and (i+1)^(th) image is v*T_(i). After aligning theblurred foreground, deblurring is performed using MID 650 by solving theEquation (10). Due to noisy alpha maps, this initial estimate ofdeblurred image is noisy and contains erroneous background contribution.

Refinement

The alignment of the foreground layers in the previous step can beslightly off, because motion may not be constant assumption in realworld scenes. The misalignment causes errors in deconvolution. We refinethe alignment 660 using the deblurred foreground f obtained in previousstep. Specifically, we blur the sharp foreground f using the blurkernels h_(k) and find the shift between the synthetic blurredforeground and f_(k) ^(b). Thus, all blurred foreground f_(k) ^(b)s arecorrelated through the sharp foreground f and can be accurately aligned.

Segmentation Refinement

Using the deblurred foreground f, refined alpha maps are computed as

α=1−(i _(k) −f*h _(k))/b.   (20)

The obtained alpha map are then deblurred using the same MID algorithmand thresholded to obtain a binary segmentation m_(k) of the object.Because the linear system is well-posed, deblurring the alpha maps givesus a close solution to the true foreground segmentation. However, simplethresholding does not result in an accurate segmentation mask, whichneeds to be improved.

We refine segmentation 670 for each motion line independently. In oneembodiment, we are using a conservative threshold, e.g., 0.65, to obtainan initial segmentation mask m⁰ smaller than the size of the object. Foreach motion line, we expand segmentation mask one pixel on each side ata time, and find the best estimate, which minimizes the joint costfunction

$\begin{matrix}{{{J(m)} = {\sum\limits_{k = 1}^{N}( {\alpha_{k} - {m*h_{k}}} )^{2}}},{m \in {\{ {0,1} \}.}}} & (21)\end{matrix}$

The refinement is repeated for the entire blur region and the bestsegmentation is chosen to minimize the error for each motion line.Typically, we search within ten pixels on each side, which results in10²=100 dilations for each scan line. Typically, the quality ofsegmentation and deblurring improves even after the first iteration. Therefinement is iterated two or three times for the final result.

Although the invention has been described by way of examples ofpreferred embodiments, it is to be understood that various otheradaptations and modifications may be made within the spirit and scope ofthe invention. Therefore, it is the object of the appended claims tocover all such variations and modifications as come within the truespirit and scope of the invention.

1. A method for reducing a blur in an image of a scene, comprising aprocessor for performing steps of the method, comprising the steps of:acquiring a set of images of the scene, wherein each image in the set ofimages includes an object having a blur associated with a point spreadfunction (PSF) forming a set of point spread functions (PSFs), whereinthe set of PSFs is suitable for null-filling operation; and invertingjointly the set of images and the set of PSFs to produce an output imagehaving a reduced blur.
 2. The method of claim 1, further comprising:acquiring the set of images with exposure times, which are non-integermultiples of each other.
 3. The method of claim 1, wherein the set ofPSFs includes a first PSF, the first PSF at a particular frequency iszero, and a second PSF, wherein the second PSF at the particularfrequency is not zero.
 4. The method of claim 1, further comprising:acquiring the set of images with a single camera.
 5. The method of claim1, further comprising: acquiring the set of images using multiplecameras running at different frame rates.
 6. The method of claim 1,wherein the PSF is a motion PSF.
 7. The method of claim 1, wherein theinverting further comprising: determining the output image according to${\hat{f} = {( {\sum\limits_{k = 1}^{N}{A_{k}^{T}A_{k}}} )^{- 1}( {\sum\limits_{k = 1}^{N}{A_{k}^{T}i_{k}}} )}},$wherein {circumflex over (f)} is a vector describing the output image,A_(k) is a circulant motion smear matrix for an image k, i_(k) is avector describing the blurred object, N is a number of images in the setof images, T is a transpose operator.
 8. The method of claim 2, furthercomprising: determining the exposure times such that a deconvolutionoperator is maximized.
 9. The method of claim 2, further comprising:determining the exposure times such that the signal to noise ismaximized.
 10. A method for reducing blur in an image, comprising aprocessor for performing steps of the method, comprising the steps of:acquiring repetitively a set of images of a moving object with a set ofexposures, wherein the exposures are non-integer multiples of eachother; estimating a point spread function (PSF) using motion vectorscomputed from images acquired with the same exposure forming a set ofpoint spread functions (PSFs); determining an alpha map for each imagein the set of images; removing a background contribution from the imagesusing the alpha map; and inverting jointly the set of images and the setof PSFs to produce an output image having a reduced blur.
 11. The methodof claim 10, further comprising: repeating the determining, theremoving, and the inverting.
 12. The method of claim 10, wherein theacquiring is performed using a camera having an auto exposurebracketing.
 13. The method of claim 10, wherein for each image furthercomprising: estimating a background of the image; and generating a crudetrimap by thresholding a difference between the image and thebackground.
 14. The method of claim 10, further comprising: aligning theobject in the set of images.
 15. The method of claim 10, furthercomprising: refining a segmentation of the object for each motion lineindependently.
 16. A camera configured for reducing blur in an image,comprising: means for acquiring a set of images of a moving object,wherein each image is associated with a point spread function (PSF)forming a set of point spread functions (PSFs), wherein the set of PSFsis suitable for null-filling operation; and means for inverting jointlythe set of images and the set of PSFs to produce an output image havinga reduced blur.
 17. The camera of claim 16, wherein the set of PSFsincludes a first PSF, the first PSF at a particular frequency is zero,and a second PSF, wherein the second PSF at the particular frequency isnot zero.
 18. The camera of claim 16, wherein the means for invertingfurther comprising: means for determining the output image according to${\hat{f} = {( {\sum\limits_{k = 1}^{N}{A_{k}^{T}A_{k}}} )^{- 1}( {\sum\limits_{k = 1}^{N}{A_{k}^{T}i_{k}}} )}},$wherein {circumflex over (f)} is a vector describing the output image,A_(k) is a circulant motion smear matrix for an image k, i_(k) is avector describing the blurred object, N is a number of images in the setof images, T is a transpose operator.
 19. The camera of claim 16,further comprising: means for estimating the set of PSFs.
 20. The cameraof claim 16, further comprising: means for segmenting the set of images.